feat: add support for better handling with large arrays #402

wilkmaia · 2022-04-06T18:26:39Z

Intro

This PR is a tentative approach at improving the behavior described in fastify/fastify#3498. In short, for very large arrays (approx over 20k elements) the overhead introduced by fast-json-stringify becomes relevant and makes the stringifying process longer.

Closes fastify/fastify#3498.

Analysis

Some simple benchmarks were run both in order to better understand the issue and to understand the impact of the various approaches attempted at improving it.

Methodology

A variable-length array with similar spec as to the large one available in the linked issue was stringified using two approaches:

JSON.stringify as baseline
fast-json-stringify

Data on array length and time taken for the process to complete was collected and plotted over several different graphs, displayed below. JSON.stringify's time to complete is plotted in orange and fast-json-stringify's in blue. Time is displayed in nanoseconds. Data for each array length was averaged over 30 different measurements.

The benchmark script used (as well as helpers and dependencies) is available at https://gist.github.com/wilkmaia/c2a43a145fcd0f9dda06710933c26850.

Results

Baseline Data

Baseline data was initially collected for future comparison. It's shown that the time the two functions take to work over the data starts to get significantly discrepant with arrays of length somewhere between 16k and 32k.

Closer Look

Looking at data over a smaller interval (length of 5000 to 32000) one can see that the discrepancy starts getting more significant for arrays with about 20k items or more. In the following sections we'll consider large array an array with 20000 or more elements.

Approach 1

Array.join instead of string concatenation for large arrays.

This approach compromises performance to maintain high utility. Performance gain for the largest array tested was of about 33% and there's no loss in the lib's feature set. The performance gain comes from the fact that, for large arrays, currently, Array.join performs better than concatenating strings in v8 [1]. Browser performance apparently has been varying over the last years.

Approach 2

Array.join instead of string concatenation for all arrays.

This was the worst approach of all, performance-wise. With it the worst case scenario ran only about 13% faster. That was, though, expected, given performance for Array.join is significantly worse for smaller arrays. There's no point in following this approach, given it gives no benefit over (Approach 1)[approach-1].

Approach 3

JSON.stringify all arrays.

This approach focuses on maximizing performance to the detriment of utility. With that the lib loses spec validation for arrays and its elements but the stringify process completes quickly. Performance gain was about 77% for the worst case scenario, resulting in overall performance extremely similar to JSON.stringify for large arrays and quicker for small ones.

Approach 4

JSON.stringify for large arrays.

This approach compromises a bit of the performance gained from Approach 3 in order to provide more utility to the lib's users. Only large arrays are submitted to JSON.stringify. All others would still benefit from spec validation. Performance gain for this approach was of about 76% for the worst case scenario.

Conclusion

In the image above there's a compiled summary of the baselines plus the fast-json-stringify's updated approaches results, disregarding the smallest datasets (less than 2048 elements).

Judging by that I believe the best solution would be to either implement Approach 1 or Approach 4 or, the proposed solution in this PR, to add support for the user to select whether they want the original implementation or one of the two proposed variants.

References

[1]: Simple benchmark for string concatenation vs Array.join

Project Benchmark

Previous approach

Output for npm run benchmark before changes:

> fast-json-stringify@3.0.3 benchmark
> node bench.js

FJS creation x 4,194 ops/sec ±2.72% (88 runs sampled)
CJS creation x 99,109 ops/sec ±1.85% (85 runs sampled)
AJV Serialize creation x 40,912,505 ops/sec ±0.18% (94 runs sampled)
JSON.stringify array x 3,560 ops/sec ±1.97% (94 runs sampled)
fast-json-stringify array x 4,139 ops/sec ±0.63% (94 runs sampled)
compile-json-stringify array x 4,018 ops/sec ±1.91% (89 runs sampled)
AJV Serialize array x 4,067 ops/sec ±0.42% (94 runs sampled)
JSON.stringify long string x 11,016 ops/sec ±1.12% (94 runs sampled)
fast-json-stringify long string x 11,035 ops/sec ±1.10% (93 runs sampled)
compile-json-stringify long string x 11,084 ops/sec ±1.03% (94 runs sampled)
AJV Serialize long string x 10,249 ops/sec ±0.74% (93 runs sampled)
JSON.stringify short string x 7,642,078 ops/sec ±0.32% (95 runs sampled)
fast-json-stringify short string x 19,330,903 ops/sec ±1.35% (93 runs sampled)
compile-json-stringify short string x 19,943,259 ops/sec ±0.35% (94 runs sampled)
AJV Serialize short string x 19,143,538 ops/sec ±2.03% (94 runs sampled)
JSON.stringify obj x 2,007,192 ops/sec ±0.11% (91 runs sampled)
fast-json-stringify obj x 6,912,154 ops/sec ±2.20% (91 runs sampled)
compile-json-stringify obj x 9,561,572 ops/sec ±0.41% (92 runs sampled)
AJV Serialize obj x 5,861,766 ops/sec ±0.99% (94 runs sampled)
JSON stringify date x 525,675 ops/sec ±0.68% (94 runs sampled)
fast-json-stringify date format x 1,309,404 ops/sec ±0.83% (94 runs sampled)
compile-json-stringify date format x 527,306 ops/sec ±0.77% (92 runs sampled)

New approach

Output for npm run benchmark after changes:

> fast-json-stringify@3.0.3 benchmark
> node bench.js

FJS creation x 4,065 ops/sec ±2.84% (82 runs sampled)
CJS creation x 98,789 ops/sec ±1.34% (88 runs sampled)
AJV Serialize creation x 30,201,125 ops/sec ±1.45% (81 runs sampled)
JSON.stringify array x 3,128 ops/sec ±1.72% (95 runs sampled)
fast-json-stringify array default x 4,567 ops/sec ±0.82% (93 runs sampled)
fast-json-stringify array json-stringify x 4,562 ops/sec ±1.61% (93 runs sampled)
fast-json-stringify array array-join x 4,623 ops/sec ±0.31% (93 runs sampled)
compile-json-stringify array x 4,051 ops/sec ±2.08% (89 runs sampled)
AJV Serialize array x 4,032 ops/sec ±0.20% (96 runs sampled)
JSON.stringify large array x 157 ops/sec ±0.73% (86 runs sampled)
fast-json-stringify large array default x 48.72 ops/sec ±4.92% (48 runs sampled)
fast-json-stringify large array json-stringify x 157 ops/sec ±0.76% (86 runs sampled)
fast-json-stringify large array array-join x 69.04 ops/sec ±4.47% (53 runs sampled)
compile-json-stringify large array x 175 ops/sec ±4.47% (79 runs sampled)
AJV Serialize large array x 58.76 ops/sec ±4.59% (60 runs sampled)
JSON.stringify long string x 11,112 ops/sec ±0.22% (92 runs sampled)
fast-json-stringify long string x 11,053 ops/sec ±1.35% (91 runs sampled)
compile-json-stringify long string x 11,113 ops/sec ±0.22% (92 runs sampled)
AJV Serialize long string x 10,148 ops/sec ±1.59% (91 runs sampled)
JSON.stringify short string x 7,654,762 ops/sec ±0.18% (94 runs sampled)
fast-json-stringify short string x 18,895,714 ops/sec ±2.06% (90 runs sampled)
compile-json-stringify short string x 15,956,471 ops/sec ±1.75% (92 runs sampled)
AJV Serialize short string x 15,845,197 ops/sec ±0.84% (90 runs sampled)
JSON.stringify obj x 1,964,339 ops/sec ±2.23% (93 runs sampled)
fast-json-stringify obj x 6,972,524 ops/sec ±0.40% (94 runs sampled)
compile-json-stringify obj x 9,403,372 ops/sec ±1.99% (92 runs sampled)
AJV Serialize obj x 5,933,215 ops/sec ±0.83% (91 runs sampled)
JSON stringify date x 523,850 ops/sec ±0.53% (93 runs sampled)
fast-json-stringify date format x 1,297,452 ops/sec ±0.89% (94 runs sampled)
compile-json-stringify date format x 548,051 ops/sec ±1.16% (93 runs sampled)

Checklist

run npm run test and npm run benchmark
tests and/or benchmarks are included
documentation is changed or added
commit message and code follows the Developer's Certification of Origin
and the Code of conduct

README.md

simoneb · 2022-04-06T19:31:15Z

A couple of things:

how did you run these benchmarks? I would be useful to include the code to run them so other people can do the same autonomously
it would have been really useful if the charts were using a log scale instead of linear. in the latter case it's really hard to spot differences in the performance of the various approaches for small arrays

mcollina

Really solid work! We can include this in Fastify v4.

xtx1130 · 2022-04-07T08:47:06Z

for discussion only, I try to change this two const in bench.js :

const LARGE_ARRAY_SIZE = Math.pow(2,14)
const MULTI_ARRAY_LENGHT = Math.pow(2,10)

and the answer is :

FJS creation x 8,059 ops/sec ±17.29% (88 runs sampled)
CJS creation x 143,612 ops/sec ±0.41% (92 runs sampled)
AJV Serialize creation x 64,358,288 ops/sec ±0.59% (96 runs sampled)
JSON.stringify array x 5,771 ops/sec ±0.74% (99 runs sampled)
fast-json-stringify array default x 7,984 ops/sec ±1.29% (97 runs sampled)
fast-json-stringify array json-stringify x 7,263 ops/sec ±0.32% (98 runs sampled)
fast-json-stringify array array-join x 6,999 ops/sec ±0.88% (96 runs sampled)
compile-json-stringify array x 7,810 ops/sec ±0.38% (97 runs sampled)
AJV Serialize array x 7,852 ops/sec ±0.17% (97 runs sampled)
JSON.stringify large array x 365 ops/sec ±0.14% (93 runs sampled)
fast-json-stringify large array default x 357 ops/sec ±0.18% (71 runs sampled)
fast-json-stringify large array json-stringify x 326 ops/sec ±0.12% (74 runs sampled)
fast-json-stringify large array array-join x 316 ops/sec ±0.57% (72 runs sampled)
compile-json-stringify large array x 310 ops/sec ±1.16% (65 runs sampled)
AJV Serialize large array x 402 ops/sec ±0.09% (94 runs sampled)
JSON.stringify long string x 11,856 ops/sec ±0.37% (96 runs sampled)
fast-json-stringify long string x 11,934 ops/sec ±0.18% (98 runs sampled)
compile-json-stringify long string x 11,918 ops/sec ±0.24% (95 runs sampled)
AJV Serialize long string x 22,056 ops/sec ±0.15% (98 runs sampled)
JSON.stringify short string x 12,208,053 ops/sec ±0.13% (95 runs sampled)
fast-json-stringify short string x 38,396,194 ops/sec ±3.45% (88 runs sampled)
compile-json-stringify short string x 30,411,984 ops/sec ±1.07% (96 runs sampled)
AJV Serialize short string x 37,409,999 ops/sec ±0.54% (93 runs sampled)
JSON.stringify obj x 3,057,954 ops/sec ±1.76% (93 runs sampled)
fast-json-stringify obj x 12,644,309 ops/sec ±0.28% (96 runs sampled)
compile-json-stringify obj x 19,205,007 ops/sec ±0.69% (97 runs sampled)
AJV Serialize obj x 10,518,552 ops/sec ±1.19% (92 runs sampled)
JSON stringify date x 1,135,091 ops/sec ±0.41% (98 runs sampled)
fast-json-stringify date format x 1,995,636 ops/sec ±0.27% (90 runs sampled)
compile-json-stringify date format x 1,125,693 ops/sec ±0.56% (97 runs sampled)

you can see:

fast-json-stringify large array default x 357 ops/sec ±0.18% (71 runs sampled)
fast-json-stringify large array json-stringify x 326 ops/sec ±0.12% (74 runs sampled)
fast-json-stringify large array array-join x 316 ops/sec ±0.57% (72 runs sampled)

The default is faster, But I don't know why……

update: test in nodejs v16.14.2

mcollina

can you add a test for all if those settings?

README.md

wilkmaia · 2022-04-12T12:23:33Z

@xtx1130 I believe it's due to the choice of LARGE_ARRAY_SIZE you've made. You see, 2^14 is 16384. We only start noticing discrepant results for arrays with around 20000 elements. If you set your LARGE_ARRAY_SIZE to be 2^15 (or a bigger exponent) you'll likely find similar results. Can you test it and let us know if this is not true?

wilkmaia · 2022-04-12T13:34:05Z

@simoneb

how did you run these benchmarks? I would be useful to include the code to run them so other people can do the same autonomously

It was a (somewhat messy by the end) script I've made. I'll tidy it up and add a gist to it to the PR's description. I don't think it should be added to the code base, given we already have a benchmark suite set up.

it would have been really useful if the charts were using a log scale instead of linear. in the latter case it's really hard to spot differences in the performance of the various approaches for small arrays

I see your point. Indeed that'd help spot differences in the smaller ranges but, at the same time, given the overall graph scale, it'd make differences harder to spot in the larger ranges. Visually we'd see small changes but those changes would be much more meaningful than in the smaller ranges.

And, since this is really only an issue for very large arrays, I compromised on displaying actual time differences instead of a log-scaled version of it so we could have a more direct and easier to read result.

For instance, the data displayed in the Conclusion section would be like this on a log-10 scale:

mcollina

lgtm

wilkmaia · 2022-04-12T14:41:34Z

@simoneb added the scripts used to the PR description (https://gist.github.com/wilkmaia/c2a43a145fcd0f9dda06710933c26850) under Analysis.Methodology.

index.js

xtx1130 · 2022-04-13T03:22:47Z

@xtx1130 I believe it's due to the choice of LARGE_ARRAY_SIZE you've made. You see, 2^14 is 16384. We only start noticing discrepant results for arrays with around 20000 elements. If you set your LARGE_ARRAY_SIZE to be 2^15 (or a bigger exponent) you'll likely find similar results. Can you test it and let us know if this is not true?

Case 1:

const LARGE_ARRAY_SIZE = Math.pow(2,15)

result 1:

fast-json-stringify large array default x 168 ops/sec ±1.81% (85 runs sampled)
fast-json-stringify large array json-stringify x 180 ops/sec ±0.58% (84 runs sampled)
fast-json-stringify large array array-join x 78.31 ops/sec ±2.93% (59 runs sampled)

result 2:

fast-json-stringify large array default x 176 ops/sec ±0.28% (90 runs sampled)
fast-json-stringify large array json-stringify x 179 ops/sec ±0.81% (83 runs sampled)
fast-json-stringify large array array-join x 77.49 ops/sec ±3.38% (60 runs sampled)

Case 2:

const LARGE_ARRAY_SIZE = 20000

result 1:

fast-json-stringify large array default x 124 ops/sec ±0.55% (78 runs sampled)
fast-json-stringify large array json-stringify x 298 ops/sec ±0.15% (92 runs sampled)
fast-json-stringify large array array-join x 137 ops/sec ±2.38% (58 runs sampled)

result 2:

fast-json-stringify large array default x 122 ops/sec ±0.90% (51 runs sampled)
fast-json-stringify large array json-stringify x 296 ops/sec ±0.34% (94 runs sampled)
fast-json-stringify large array array-join x 140 ops/sec ±1.94% (59 runs sampled)

There is a very big difference between array-join and default in different case but all > 20000.
Between default and json-stringify the gap is relatively small in case 1.

In default mode, Math.pow(2,15) has a better pref than 20000, I'm curious why this is.

wilkmaia · 2022-04-13T14:21:48Z

@xtx1130 that's indeed an odd behavior. I'm not, though, able to reproduce it. I've tested on a different system (node.js v16.14.2) and got the following results (mind default is now what used to be array-join):

LARGE_ARRAY_SIZE = Math.pow(2, 15)

I  ~/D/w/n/b/fast-json-stringify   *$…  npm run benchmark                                                                                                                                                      Wed 13 Apr 2022 11:14:04 AM -03

> fast-json-stringify@3.0.3 benchmark
> node bench.js

JSON.stringify large array x 139 ops/sec ±0.11% (79 runs sampled)
fast-json-stringify large array default x 63.04 ops/sec ±2.63% (66 runs sampled)
fast-json-stringify large array json-stringify x 139 ops/sec ±0.17% (79 runs sampled)
compile-json-stringify large array x 122 ops/sec ±0.13% (78 runs sampled)
AJV Serialize large array x 168 ops/sec ±0.11% (86 runs sampled)
I  ~/D/w/n/b/fast-json-stringify   *$…  npm run benchmark                                                                                                                                                28s  Wed 13 Apr 2022 11:14:36 AM -03

> fast-json-stringify@3.0.3 benchmark
> node bench.js

JSON.stringify large array x 139 ops/sec ±0.08% (88 runs sampled)
fast-json-stringify large array default x 62.89 ops/sec ±2.75% (66 runs sampled)
fast-json-stringify large array json-stringify x 138 ops/sec ±0.15% (80 runs sampled)
compile-json-stringify large array x 121 ops/sec ±0.17% (77 runs sampled)
AJV Serialize large array x 169 ops/sec ±0.16% (86 runs sampled)

LARGE_ARRAY_SIZE = 2e4

 I  ~/D/w/n/b/fast-json-stringify   *$…  npm run benchmark                                                                                                                                              27.8s  Wed 13 Apr 2022 11:15:13 AM -03

> fast-json-stringify@3.0.3 benchmark
> node bench.js

JSON.stringify large array x 230 ops/sec ±0.18% (89 runs sampled)
fast-json-stringify large array default x 113 ops/sec ±1.50% (65 runs sampled)
fast-json-stringify large array json-stringify x 229 ops/sec ±0.15% (89 runs sampled)
compile-json-stringify large array x 306 ops/sec ±0.15% (91 runs sampled)
AJV Serialize large array x 106 ops/sec ±0.11% (78 runs sampled)
 I  ~/D/w/n/b/fast-json-stringify   *$…  npm run benchmark                                                                                                                                              27.6s  Wed 13 Apr 2022 11:15:48 AM -03

> fast-json-stringify@3.0.3 benchmark
> node bench.js

JSON.stringify large array x 233 ops/sec ±0.10% (90 runs sampled)
fast-json-stringify large array default x 114 ops/sec ±1.42% (66 runs sampled)
fast-json-stringify large array json-stringify x 229 ops/sec ±0.16% (89 runs sampled)
compile-json-stringify large array x 306 ops/sec ±0.13% (91 runs sampled)
AJV Serialize large array x 104 ops/sec ±0.21% (77 runs sampled)

As expected, the higher the value of LARGE_ARRAY_SIZE, the less ops/sec we got throughout all tests.

Is it possible you had fluctuating CPU usage on your system during those tests? Maybe some process in the background or things like video/audio processing were running alongside the tests. That could explain the discrepancy, at some level. Besides that, with only the info we have, I don't have many ideas as of why you're experiencing these different results. If you have access to a different system, you could maybe check if you can reproduce that behavior there as well. Finally, I'm assuming you're running the benchmark (npm run benchmark) with no changes to the source code. If that's not the case, this could also explain the behavior.

climba03003 · 2022-04-13T14:54:10Z

Is it possible you had fluctuating CPU usage on your system during those tests?

I get the same result as @xtx1130, the default / no-op is way faster / consistence than array-join in LARGE_ARRAY_SIZE = Math.pow(2, 15).

I am not sure if it is affected by the environment. Here is the system information.

$ node -v
v16.14.0

$ uname -a
Linux <REDACTED> 4.4.0-19041-Microsoft #1237-Microsoft Sat Sep 11 14:32:00 PST 2021 x86_64 x86_64 x86_64 GNU/Linux

$ lshw -short
H/W path  Device  Class      Description
========================================
                  system     Computer
/0                bus        Motherboard
/0/409            generic
/0/0              memory     31GiB System memory
/0/1              processor  AMD Ryzen 5 3600 6-Core Processor

wilkmaia · 2022-04-13T17:10:32Z

@climba03003 @xtx1130 when you have some time can you please re-run the benchmark using LARGE_ARRAY_SIZE = Math.pow(2, 18) ?

climba03003 · 2022-04-13T17:57:52Z

when you have some time can you please re-run the benchmark using LARGE_ARRAY_SIZE = Math.pow(2, 18) ?

@wilkmaia Here is the multiple benchmark I run to figure out which number is actually dropping. You may check the last one 35000~40000 - 1000 each step, it seems like no-op start droping after 39000 and array-join out perform after 40000.

To get everyone in sync the no-op actually the default value of first implementation in this PR.
See the below code:

switch (largeArrayMechanism) {
  case 'default':
    break

  case 'json-stringify':
    code += `
    return JSON.stringify(obj)`
    break

  case 'array-join':
    code += `
    return \`[\${obj.map(${result.mapFnName}).join(',')}]\``
    break

  default:
    throw new Error(`Unsupported large array mechanism ${largeArrayMechanism}`)
}

Math.pow(2, 18) - `array-join`(~4.6 ops/sec) better than `no-op`(~3.1 ops/sec)

Check Result

$ node bench.js 
LARGE_ARRAY_SIZE  262144
JSON.stringify array x 4,496 ops/sec ±0.29% (98 runs sampled)
fast-json-stringify array default x 7,563 ops/sec ±0.68% (95 runs sampled)
fast-json-stringify array json-stringify x 7,887 ops/sec ±0.42% (95 runs sampled)
fast-json-stringify array array-join x 7,801 ops/sec ±1.52% (94 runs sampled)
compile-json-stringify array x 6,855 ops/sec ±1.78% (91 runs sampled)
JSON.stringify large array x 16.34 ops/sec ±1.95% (45 runs sampled)
fast-json-stringify large array default x 3.21 ops/sec ±6.07% (13 runs sampled)
fast-json-stringify large array json-stringify x 17.01 ops/sec ±0.22% (46 runs sampled)
fast-json-stringify large array array-join x 4.62 ops/sec ±8.50% (16 runs sampled)
compile-json-stringify large array x 3.69 ops/sec ±8.28% (14 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  262144
JSON.stringify array x 4,407 ops/sec ±0.83% (96 runs sampled)
fast-json-stringify array default x 7,380 ops/sec ±1.12% (93 runs sampled)
fast-json-stringify array json-stringify x 7,835 ops/sec ±0.35% (97 runs sampled)
fast-json-stringify array array-join x 7,476 ops/sec ±1.50% (96 runs sampled)
compile-json-stringify array x 7,262 ops/sec ±1.32% (88 runs sampled)
JSON.stringify large array x 16.99 ops/sec ±0.35% (46 runs sampled)
fast-json-stringify large array default x 3.13 ops/sec ±8.97% (12 runs sampled)
fast-json-stringify large array json-stringify x 16.68 ops/sec ±0.91% (45 runs sampled)
fast-json-stringify large array array-join x 4.61 ops/sec ±8.39% (17 runs sampled)
compile-json-stringify large array x 3.64 ops/sec ±7.71% (14 runs sampled)

Math.pow(2, 17) - `array-join`(~10 ops/sec) better than `no-op`(~7 ops/sec)

Check Result

$ node bench.js 
LARGE_ARRAY_SIZE  131072
JSON.stringify array x 4,429 ops/sec ±0.62% (94 runs sampled)
fast-json-stringify array default x 7,808 ops/sec ±0.86% (94 runs sampled)
fast-json-stringify array json-stringify x 7,433 ops/sec ±0.44% (95 runs sampled)
fast-json-stringify array array-join x 7,002 ops/sec ±0.38% (97 runs sampled)
compile-json-stringify array x 7,753 ops/sec ±0.82% (91 runs sampled)
JSON.stringify large array x 33.85 ops/sec ±0.33% (59 runs sampled)
fast-json-stringify large array default x 7.05 ops/sec ±6.41% (22 runs sampled)
fast-json-stringify large array json-stringify x 33.83 ops/sec ±0.32% (60 runs sampled)
fast-json-stringify large array array-join x 10.78 ops/sec ±7.24% (30 runs sampled)
compile-json-stringify large array x 7.80 ops/sec ±6.03% (24 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  131072
JSON.stringify array x 4,408 ops/sec ±0.86% (95 runs sampled)
fast-json-stringify array default x 7,477 ops/sec ±1.11% (92 runs sampled)
fast-json-stringify array json-stringify x 7,238 ops/sec ±0.92% (93 runs sampled)
fast-json-stringify array array-join x 7,222 ops/sec ±0.96% (93 runs sampled)
compile-json-stringify array x 7,564 ops/sec ±0.93% (93 runs sampled)
JSON.stringify large array x 33.75 ops/sec ±0.34% (59 runs sampled)
fast-json-stringify large array default x 6.82 ops/sec ±5.79% (22 runs sampled)
fast-json-stringify large array json-stringify x 33.78 ops/sec ±0.19% (59 runs sampled)
fast-json-stringify large array array-join x 10.02 ops/sec ±7.09% (30 runs sampled)
compile-json-stringify large array x 7.64 ops/sec ±6.81% (24 runs sampled)

Math.pow(2, 16) - `array-join`(~23 ops/sec) better than `no-op`(~17.5 ops/sec)

Check Result

$ node bench.js 
LARGE_ARRAY_SIZE  65536
JSON.stringify array x 4,496 ops/sec ±0.34% (97 runs sampled)
fast-json-stringify array default x 7,565 ops/sec ±0.30% (94 runs sampled)
fast-json-stringify array json-stringify x 7,213 ops/sec ±0.52% (94 runs sampled)
fast-json-stringify array array-join x 7,566 ops/sec ±1.23% (94 runs sampled)
compile-json-stringify array x 7,280 ops/sec ±1.04% (89 runs sampled)
JSON.stringify large array x 66.98 ops/sec ±0.29% (69 runs sampled)
fast-json-stringify large array default x 17.45 ops/sec ±3.96% (34 runs sampled)
fast-json-stringify large array json-stringify x 67.36 ops/sec ±0.25% (70 runs sampled)
fast-json-stringify large array array-join x 23.48 ops/sec ±4.94% (43 runs sampled)
compile-json-stringify large array x 18.94 ops/sec ±5.21% (36 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  65536
JSON.stringify array x 4,469 ops/sec ±0.48% (95 runs sampled)
fast-json-stringify array default x 7,090 ops/sec ±0.32% (94 runs sampled)
fast-json-stringify array json-stringify x 7,956 ops/sec ±0.48% (96 runs sampled)
fast-json-stringify array array-join x 7,772 ops/sec ±0.43% (96 runs sampled)
compile-json-stringify array x 7,614 ops/sec ±0.88% (91 runs sampled)
JSON.stringify large array x 66.25 ops/sec ±0.74% (69 runs sampled)
fast-json-stringify large array default x 17.51 ops/sec ±5.84% (33 runs sampled)
fast-json-stringify large array json-stringify x 66.51 ops/sec ±0.66% (69 runs sampled)
fast-json-stringify large array array-join x 23.16 ops/sec ±5.10% (43 runs sampled)
compile-json-stringify large array x 18.86 ops/sec ±5.32% (36 runs sampled)

Math.pow(2, 15) - `no-op` (~145 ops/sec) better than `array-join` (~50 ops/sec)

Check Result

$ node bench.js 
LARGE_ARRAY_SIZE  32768
JSON.stringify array x 4,479 ops/sec ±0.32% (97 runs sampled)
fast-json-stringify array default x 7,599 ops/sec ±0.74% (95 runs sampled)
fast-json-stringify array json-stringify x 7,697 ops/sec ±0.29% (95 runs sampled)
fast-json-stringify array array-join x 7,531 ops/sec ±0.31% (95 runs sampled)
compile-json-stringify array x 6,950 ops/sec ±1.56% (85 runs sampled)
JSON.stringify large array x 131 ops/sec ±1.01% (83 runs sampled)
fast-json-stringify large array default x 146 ops/sec ±1.43% (83 runs sampled)
fast-json-stringify large array json-stringify x 136 ops/sec ±0.51% (86 runs sampled)
fast-json-stringify large array array-join x 51.19 ops/sec ±4.84% (46 runs sampled)
compile-json-stringify large array x 137 ops/sec ±0.23% (80 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  32768
JSON.stringify array x 4,427 ops/sec ±0.66% (96 runs sampled)
fast-json-stringify array default x 7,676 ops/sec ±0.74% (95 runs sampled)
fast-json-stringify array json-stringify x 6,912 ops/sec ±0.71% (95 runs sampled)
fast-json-stringify array array-join x 7,368 ops/sec ±0.52% (92 runs sampled)
compile-json-stringify array x 7,622 ops/sec ±0.94% (93 runs sampled)
JSON.stringify large array x 134 ops/sec ±0.99% (85 runs sampled)
fast-json-stringify large array default x 144 ops/sec ±1.31% (81 runs sampled)
fast-json-stringify large array json-stringify x 134 ops/sec ±0.64% (86 runs sampled)
fast-json-stringify large array array-join x 50.42 ops/sec ±4.59% (46 runs sampled)
compile-json-stringify large array x 139 ops/sec ±0.69% (79 runs sampled)

20000 - `no-op` (~93 ops/sec) nearly the same `array-join` (~93 ops/sec)

Check Result

$ node bench.js 
LARGE_ARRAY_SIZE  20000
JSON.stringify array x 4,436 ops/sec ±0.52% (96 runs sampled)
fast-json-stringify array default x 7,707 ops/sec ±0.60% (94 runs sampled)
fast-json-stringify array json-stringify x 7,641 ops/sec ±0.55% (95 runs sampled)
fast-json-stringify array array-join x 7,598 ops/sec ±0.36% (93 runs sampled)
compile-json-stringify array x 7,624 ops/sec ±0.65% (93 runs sampled)
JSON.stringify large array x 218 ops/sec ±1.08% (85 runs sampled)
fast-json-stringify large array default x 96.91 ops/sec ±0.43% (82 runs sampled)
fast-json-stringify large array json-stringify x 222 ops/sec ±0.92% (87 runs sampled)
fast-json-stringify large array array-join x 93.16 ops/sec ±3.35% (55 runs sampled)
compile-json-stringify large array x 329 ops/sec ±1.08% (87 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  20000
JSON.stringify array x 4,433 ops/sec ±0.86% (95 runs sampled)
fast-json-stringify array default x 7,322 ops/sec ±1.18% (92 runs sampled)
fast-json-stringify array json-stringify x 7,225 ops/sec ±1.41% (91 runs sampled)
fast-json-stringify array array-join x 7,377 ops/sec ±0.93% (93 runs sampled)
compile-json-stringify array x 7,368 ops/sec ±1.04% (93 runs sampled)
JSON.stringify large array x 223 ops/sec ±0.49% (87 runs sampled)
fast-json-stringify large array default x 91.85 ops/sec ±1.33% (78 runs sampled)
fast-json-stringify large array json-stringify x 218 ops/sec ±1.04% (85 runs sampled)
fast-json-stringify large array array-join x 94.76 ops/sec ±3.59% (55 runs sampled)
compile-json-stringify large array x 345 ops/sec ±0.42% (91 runs sampled)

35000~40000 - 1000 each step

Check Result

$ node bench.js 
LARGE_ARRAY_SIZE  35000
JSON.stringify array x 4,477 ops/sec ±0.27% (97 runs sampled)
fast-json-stringify array default x 7,789 ops/sec ±0.76% (96 runs sampled)
fast-json-stringify array json-stringify x 7,724 ops/sec ±0.63% (95 runs sampled)
fast-json-stringify array array-join x 7,949 ops/sec ±0.79% (96 runs sampled)
compile-json-stringify array x 7,610 ops/sec ±0.64% (95 runs sampled)
JSON.stringify large array x 127 ops/sec ±0.69% (80 runs sampled)
fast-json-stringify large array default x 163 ops/sec ±0.99% (83 runs sampled)
fast-json-stringify large array json-stringify x 123 ops/sec ±1.13% (79 runs sampled)
fast-json-stringify large array array-join x 45.62 ops/sec ±4.78% (49 runs sampled)
compile-json-stringify large array x 144 ops/sec ±1.28% (82 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  36000
JSON.stringify array x 4,473 ops/sec ±0.26% (97 runs sampled)
fast-json-stringify array default x 7,399 ops/sec ±0.81% (93 runs sampled)
fast-json-stringify array json-stringify x 7,887 ops/sec ±0.47% (96 runs sampled)
fast-json-stringify array array-join x 7,584 ops/sec ±0.41% (96 runs sampled)
compile-json-stringify array x 7,113 ops/sec ±0.99% (90 runs sampled)
JSON.stringify large array x 124 ops/sec ±0.44% (79 runs sampled)
fast-json-stringify large array default x 171 ops/sec ±0.54% (87 runs sampled)
fast-json-stringify large array json-stringify x 123 ops/sec ±0.37% (79 runs sampled)
fast-json-stringify large array array-join x 46.32 ops/sec ±4.72% (53 runs sampled)
compile-json-stringify large array x 155 ops/sec ±0.44% (87 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  37000
JSON.stringify array x 4,505 ops/sec ±0.21% (97 runs sampled)
fast-json-stringify array default x 7,729 ops/sec ±1.13% (95 runs sampled)
fast-json-stringify array json-stringify x 7,490 ops/sec ±1.34% (93 runs sampled)
fast-json-stringify array array-join x 7,889 ops/sec ±0.41% (97 runs sampled)
compile-json-stringify array x 7,699 ops/sec ±0.43% (91 runs sampled)
JSON.stringify large array x 120 ops/sec ±0.36% (77 runs sampled)
fast-json-stringify large array default x 183 ops/sec ±0.45% (85 runs sampled)
fast-json-stringify large array json-stringify x 120 ops/sec ±1.03% (77 runs sampled)
fast-json-stringify large array array-join x 43.26 ops/sec ±4.69% (47 runs sampled)
compile-json-stringify large array x 157 ops/sec ±0.61% (80 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  38000
JSON.stringify array x 4,489 ops/sec ±0.29% (98 runs sampled)
fast-json-stringify array default x 7,249 ops/sec ±0.85% (92 runs sampled)
fast-json-stringify array json-stringify x 7,673 ops/sec ±0.78% (90 runs sampled)
fast-json-stringify array array-join x 7,496 ops/sec ±0.99% (95 runs sampled)
compile-json-stringify array x 7,570 ops/sec ±0.76% (92 runs sampled)
JSON.stringify large array x 117 ops/sec ±0.56% (84 runs sampled)
fast-json-stringify large array default x 181 ops/sec ±0.49% (84 runs sampled)
fast-json-stringify large array json-stringify x 112 ops/sec ±1.63% (81 runs sampled)
fast-json-stringify large array array-join x 39.88 ops/sec ±4.19% (46 runs sampled)
compile-json-stringify large array x 133 ops/sec ±2.52% (63 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  39000
JSON.stringify array x 4,414 ops/sec ±1.07% (93 runs sampled)
fast-json-stringify array default x 7,815 ops/sec ±0.70% (94 runs sampled)
fast-json-stringify array json-stringify x 7,739 ops/sec ±0.54% (96 runs sampled)
fast-json-stringify array array-join x 7,556 ops/sec ±0.61% (94 runs sampled)
compile-json-stringify array x 6,887 ops/sec ±0.73% (94 runs sampled)
JSON.stringify large array x 111 ops/sec ±1.04% (81 runs sampled)
fast-json-stringify large array default x 55.91 ops/sec ±1.03% (57 runs sampled)
fast-json-stringify large array json-stringify x 113 ops/sec ±0.77% (82 runs sampled)
fast-json-stringify large array array-join x 39.07 ops/sec ±5.20% (53 runs sampled)
compile-json-stringify large array x 156 ops/sec ±0.43% (88 runs sampled)
$ node bench.js
LARGE_ARRAY_SIZE  40000
JSON.stringify array x 4,458 ops/sec ±0.31% (97 runs sampled)
fast-json-stringify array default x 7,771 ops/sec ±1.38% (95 runs sampled)
fast-json-stringify array json-stringify x 7,861 ops/sec ±0.43% (95 runs sampled)
fast-json-stringify array array-join x 7,885 ops/sec ±0.38% (97 runs sampled)
compile-json-stringify array x 7,645 ops/sec ±0.59% (92 runs sampled)
JSON.stringify large array x 111 ops/sec ±0.88% (81 runs sampled)
fast-json-stringify large array default x 31.99 ops/sec ±9.14% (43 runs sampled)
fast-json-stringify large array json-stringify x 111 ops/sec ±0.44% (81 runs sampled)
fast-json-stringify large array array-join x 40.18 ops/sec ±4.32% (53 runs sampled)
compile-json-stringify large array x 176 ops/sec ±0.38% (81 runs sampled)

wilkmaia · 2022-04-13T23:15:11Z

Thanks for the detailed info, @climba03003. Apparently we're seeing some performance difference based on the host machine. That's my best guess with the info we have so far.

I don't think this changes much on what this PR is aiming at, anyway, specially with the latest changes, which included an option for the user to set how large a large array is. For instance, for your specific scenario you might want to set that to 40000 instead of the default 20000.

climba03003

LGTM.

xtx1130 · 2022-04-14T02:33:55Z

Thanks for @climba03003 's test cases, I'm +-1 of 20000. But I still have a question, why the performance of 20000 looks like unusual?

simoneb · 2022-04-15T16:04:49Z

If we're happy with the result, I think this can now be merged

wilkmaia added 2 commits April 6, 2022 14:56

feat: add support for better handling with large arrays

0ea94b2

chore: add large arrays section to readme

f44faf5

wilkmaia mentioned this pull request Apr 6, 2022

Worse performance when response schema specified fastify/fastify#3498

Closed

2 tasks

fix: fix typo on unit

fafb9f4

RafaelGSS reviewed Apr 6, 2022

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

mcollina reviewed Apr 6, 2022

View reviewed changes

mcollina reviewed Apr 7, 2022

View reviewed changes

README.md Show resolved Hide resolved

wilkmaia added 2 commits April 12, 2022 10:52

chore: set array-join as default large array mechanism

bad848a

chore: add tests for large array mechanisms

007916c

mcollina approved these changes Apr 12, 2022

View reviewed changes

wilkmaia requested a review from RafaelGSS April 12, 2022 14:41

xtx1130 suggested changes Apr 13, 2022

View reviewed changes

index.js Outdated Show resolved Hide resolved

feat: add option for the user to specify the large array size

ea87106

climba03003 approved these changes Apr 14, 2022

View reviewed changes

mcollina merged commit 66b79f0 into fastify:master Apr 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for better handling with large arrays #402

feat: add support for better handling with large arrays #402

wilkmaia commented Apr 6, 2022 •

edited

Loading

simoneb commented Apr 6, 2022

mcollina left a comment

xtx1130 commented Apr 7, 2022 •

edited

Loading

mcollina left a comment

wilkmaia commented Apr 12, 2022

wilkmaia commented Apr 12, 2022

mcollina left a comment

wilkmaia commented Apr 12, 2022

xtx1130 commented Apr 13, 2022

wilkmaia commented Apr 13, 2022

climba03003 commented Apr 13, 2022 •

edited

Loading

wilkmaia commented Apr 13, 2022 •

edited

Loading

climba03003 commented Apr 13, 2022 •

edited

Loading

wilkmaia commented Apr 13, 2022

climba03003 left a comment

xtx1130 commented Apr 14, 2022

simoneb commented Apr 15, 2022

feat: add support for better handling with large arrays #402

feat: add support for better handling with large arrays #402

Conversation

wilkmaia commented Apr 6, 2022 • edited Loading

Intro

Analysis

Methodology

Results

Baseline Data

Closer Look

Approach 1

Approach 2

Approach 3

Approach 4

Conclusion

References

Project Benchmark

Previous approach

New approach

Checklist

simoneb commented Apr 6, 2022

mcollina left a comment

Choose a reason for hiding this comment

xtx1130 commented Apr 7, 2022 • edited Loading

mcollina left a comment

Choose a reason for hiding this comment

wilkmaia commented Apr 12, 2022

wilkmaia commented Apr 12, 2022

mcollina left a comment

Choose a reason for hiding this comment

wilkmaia commented Apr 12, 2022

xtx1130 commented Apr 13, 2022

wilkmaia commented Apr 13, 2022

LARGE_ARRAY_SIZE = Math.pow(2, 15)

LARGE_ARRAY_SIZE = 2e4

climba03003 commented Apr 13, 2022 • edited Loading

wilkmaia commented Apr 13, 2022 • edited Loading

climba03003 commented Apr 13, 2022 • edited Loading

Math.pow(2, 18) - array-join(~4.6 ops/sec) better than no-op(~3.1 ops/sec)

Math.pow(2, 17) - array-join(~10 ops/sec) better than no-op(~7 ops/sec)

Math.pow(2, 16) - array-join(~23 ops/sec) better than no-op(~17.5 ops/sec)

Math.pow(2, 15) - no-op (~145 ops/sec) better than array-join (~50 ops/sec)

20000 - no-op (~93 ops/sec) nearly the same array-join (~93 ops/sec)

35000~40000 - 1000 each step

wilkmaia commented Apr 13, 2022

climba03003 left a comment

Choose a reason for hiding this comment

xtx1130 commented Apr 14, 2022

simoneb commented Apr 15, 2022

wilkmaia commented Apr 6, 2022 •

edited

Loading

xtx1130 commented Apr 7, 2022 •

edited

Loading

climba03003 commented Apr 13, 2022 •

edited

Loading

wilkmaia commented Apr 13, 2022 •

edited

Loading

climba03003 commented Apr 13, 2022 •

edited

Loading

Math.pow(2, 18) - `array-join`(~4.6 ops/sec) better than `no-op`(~3.1 ops/sec)

Math.pow(2, 17) - `array-join`(~10 ops/sec) better than `no-op`(~7 ops/sec)

Math.pow(2, 16) - `array-join`(~23 ops/sec) better than `no-op`(~17.5 ops/sec)

Math.pow(2, 15) - `no-op` (~145 ops/sec) better than `array-join` (~50 ops/sec)

20000 - `no-op` (~93 ops/sec) nearly the same `array-join` (~93 ops/sec)